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1, INTRODUCTION 

Image restoration process is an important step used in denoising that refers to the recovery of an 
underlying image from an observation that has been corrupted by noise. There are various sources let the 
digital images to be corrupted by poor contrast and noise. These sources include image transmission, 
acquisition, compression, quantization, illumination conditions, malfunctioning instruments, ill positions 
and more. The application area of such image denoising includes general object recognition, 
digital entertainment, medical image understanding and remote sensing imaging [1] and we are especially 
interested in the restoration of corrupted facial image for digital forensic applications [2]. Similarly, image 
denoising is an inverse problem of any image corruption process and can be viewed as a filtering system with 
visible fuzziness as the existence of fuzziness in the image signal and contaminated signal thus many fuzzy 
logic based approaches have shown their strengths in this domain [3]. 

In this paper, we are especially interested in the restoration of corrupted facial image from rough 
images like casual smartphone photographs and images from closed circuit television (CCTV) camera. 
In recent years, CCTV has been widely used for recording random scenes that are easy to identify suspected 
criminals and it is even well visible at night. Digital forensic concerns with an automated face recognition 
scenario that involves comparing degraded facial photographs of subjects against their high-resolution 
counterparts [4]. However, problem often encountered in forensic face recognition involves low-resolution 
face images that have been faxed, printed, or heavily compressed [5]. Therefore, it is necessary to restore the 
damaged color images from CCTV or smartphone that has been an important subsystem of any developed 
digital forensic software. 
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While previous attempts on the systematic image restoration from degraded image in digital forensic 
software are usually comparing degraded images with high-resolute saved models of ex-convicts [4-5], 
there is a growing usage of low resolute color images in reporting suspicious subject of such incidents. 
Thus, we need fast but robust automatic facial image denoising/reconstructing methodology [2]. 
A considerable number of practical applications including digital album, surveillances videos processing and 
personal authentication involve the restoration of blurred faces [6]. 

With that purpose, we take associative memory approaches as our engine for image restoration. 
Since it is aimed at storing and recalling associations among patterns (data) [7]. Among many models of 
associative memory, Fuzzy Associative memory (FAM) [8] is one of the successful implementation of such 
structures using a fuzzy Hebbian learning rule in terms of max-min or max-product compositions for the 
synthesis of its weight matrix W but it also has similar low capacity problem [9]. FAMs possess important 
advantages including noise tolerance, unlimited storage, and one pass convergence though. An important 
property, deciding FAM performance, is the ability to capture content of each pattern, and association of 
patterns [10]. 

Implicative Fuzzy Associative Memories (IFAM) is a single-layer feedforward fuzzy neural 
networks equipped with neurons that compute the maximum of a t-norm. IFAM exhibits excellent tolerance 
with respect to either positive or negative noise with optimal absolute storage capacity [11]. While FAMs 
might be used as a powerful tool for implementing fuzzy rule-based systems, the insight that FAMs are 
closely related to mathematical morphology has led to the development of new fuzzy morphological 
associative memory models and IFAM is a good example of them which is also mathematically sound 
[12-13]. Some IFAM models exhibited their usefulness in image restoration problems [14, 15]. 

In this paper, we propose a colored facial image restoration mechanism under IFAM structure with 
emphasizing the similarity measure of the patterns being considered. Previously, FAM structure was 
successfully applied to our application domain with grey model [16]. However, the general FAM structure 
tends to have too many 0’s 1n relation with the connection strength matrix and the threshold due to its max-C 
operator usage. This FAM characteristics results in frequent incomplete image restoration in real world 
applications. Thus, we adopt IFAM model instead of FAM and let the model work directly from the colored 
image for fast scalability concerns. 


2. IMAGE RESTORATION WITH PROPOSED IFAM STRUTURE 

An associative memory paradigm may be formulated as an input—output system that is able to store 
different patterns pairs. FAM models are classified into two categories—auto-associative and hetero- 
associative. Hetero-associative FAM model has similar structure with Bidirectional Associative Memory [17] 
that has typical structure shown as Figure 1. In this scheme, the retrieved pattern is different from the input 
pattern not only in content but possibly also in type and format. 
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Figure |. Hetero-associative fuzzy associative memory structure 


In this paper, we adopt auto-associative FAM model that retrieves a previously stored pattern which 
most closely resembles the current pattern. Figure 2 shows a typical structure of FAM model. 
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Figure 2. Auto-associative fuzzy associative memory structure 


The generalized FAM (GFAM) [18], can be described in terms of the following relationship 
between an input pattern x€ [0, 1]” and the corresponding output pattern ye [0, 1]” as shown in formula (1). 


Y = (W°,-X)°0 (1) 


where Y denote the output and W denote the connection strength matrix with threshold @. @ is defined as 
Equation (2). 


6=NR=y',0<0<1 (2) 


where y* denotes the output of k” learned pattern (maximum p patterns available). 

In Equation (1), the symbol °7 denotes the max-C product where C is a t-norm that satisfies operator 
associativity, commutativity, and monotone non-decreasing [11]. And For A € [0, 1]”” and Be[0, 1]?*” 
then the max-C product C = A ° B 1s defined as Equation (3) 


Cg Vi ag C Gig Dis Vi Sit, Vi = 1 angn (3) 


However, during the image restoration process with this GFAM structure, the result output obtained 
from the Equation (1) falls into 0’s more than desired due to the max-C operator characteristics as shown in 
Equation (2). Thus, we adopt IFAM structure with colored pixel control as shown in Equation (4). 


W = (Y°7x79"), Y = (W2px™9 VO (4) 


where x’? denote the color information of the pixel. In deciding W, we take Min operation and the output 
Y 1s computed by Max operation. 

Based on this IFAM structure, we add pattern similarity to compute the final output. If there is a 
noise in the input pattern, the restoration process is based on the similarity measure shown as Equation (5). 


similarity = pas | (5) 





where S and S’ denote the stored pattern and input pattern with noise in respectively. 
Thus, the final output Y is computed by Equation (6). 


Y=S'vWvé (6) 


In order to evaluate the correctness of restored pixel, we take mean square error (MSE) measure as 
shown in Equation (7). 


n ey ay 
MSE = dizi Vi-Pi)* (7) 


n=—Z 


where y; denotes the original pixel and ¥; is the degraded pixel. The degree of freedom is n-2. 
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3. EXPERIMENT 

The proposed method was implemented in C# under Visual Studio 2017 environment with Intel (R) 
Dual Core(TM) 13-5005U CPU @ 2.0 GHz and 4 GB RAM PC. We tested 20 facial images and some 
successful and failed examples are shown in Figure 3 and Figure 4 respectively. 





(a) Original Image (b) Damaged input (c) Restored result 


Figure 3. Successful image restoration by the proposed IFAM (MSE = 0.0) 
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(a) Original Image (b) Damaged input (c) Restored result 
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Figure 4. Failed image restoration by the proposed IFAM (MSE = 0.06) 


As shown in Figure 3, we make some damage on the original image (Figure 3(a)) and make the 
inout like Figure 3(b) that is an extreme example of ststematic deletion of non-neglisable part of the image. 
The proposed IFAM was successful to restore the lost part as shown in Figure 3(c) whose MSE value 
convergers to zero. That result is actually partly helped by the symmetricity of the facial image. 
However, face recognition from such kind of partly torn photograph occurs in the real world application. 

In Figure 4, though, we have the only failed restoration case from our experiment. In that case, 
there exists non-neglisable size of mosaic area in the damaged input (Figure 4(b)). In such a case, the pixel in 
the mosaic area only had the averaged color information thus it was not sufficient for IFAM’s Max-Min 
operation to restore the lost information. Thus, that lack of information results in the failed case as shown in 
Figure 4(c) where MSE is 0.06. Still, in our experiment, MSE converges to zero in the rest 19 systematically 
or tandomly damaged photographs. For the record, the average MSE using the grey mode IFAM [16] was 
0.21 on the same set of examples thus the proposed method shows a big improvement. 

Other than that one case, the efficacy of the proposed method can be demonstrated as shown in 
Figure 5 that directly compares the result with the previous grey GFAM structure used in [16]. 
One can clearly see that the proposed method successfully restores the previously failed noise. 
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Figure 5. Comparing Facial image restoration with previous grey mode IFAM [16] 
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The reason of failure by the previous attempt shown in Figure 5(a) is that as mentioned in Section 2, 
Equation (1) converges to zero more often when the object was too smaller compared with the background in 
the process of Max-Min operation. The proposed similarity enhanced IFAM overcomes that problem as 
shown in Figure 5(b). 


4. CONCLUSION 

In this paper, we extend the IFAM model wirh enhanced similarity measure to the colored facial 
image restoration cases. Various types of FAM models have been widely applied in many engineering 
applications in the last two decades and there are different models to adapt the application areas’ 
characteristics for tasks such as estimation, prediction and inference. What we are interested in using this 
associative memiry model is to develop a subsystem in a digital forensic system with an automated face 
recognition scenario that involves comparing degraded facial photographs of subjects against their high- 
resolution counterparts from casual smartphone photographs and closed circuit television (CCTV) camera. 
Also, we need a fast, robust, and scalable method to do it. IFAM models are strongly noise tolerant but have 
zero-convergence problem in Min-Max operation. Out similarity enhanced IFAM structure was designed to 
avoid that zero-convergence problem and the proposed method was highly successful (19 out of 20 tested 
cases or 95% success rate) in experiment. 
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